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Motivation 

The advanced nature of "search" has facilitated the movement from keyword match to the 
delivery of every conceivable information topic from career, commerce, entertainment, learning... 
the list is infinite. At NASA Johnson Space Center (JSC ) the Search interface is an important 
means of knowledge transfer. By indexing multiple sources between directorates and 
organizations, the system's potential is culture changing in that through search, knowledge of the 
unique accomplishments in engineering and science can be seamlessly passed between 
generations. 

This paper reports the findings of an initial survey, the first of a four part study to help determine 
user sentiment on the intranet, or local (JSC) enterprise search environment as well as the larger 
NASA enterprise. The survey is a means through which end users provide direction on the 
development and transfer of knowledge by way of the search experience. The ideal is to identify 
what is working and what needs to be improved from the users’ vantage point by documenting: 
Where users are satisfied/dissatisfied 
Perceived value of interface components 
Gaps which cause any disappointment in search experience. 

The near term goal is it to inform JSC search in order to improve users’ ability to utilize existing 
services and infrastructure to perform tasks with a shortened life cycle. 

Continuing steps include an agency based focus with modified questions to accomplish a similar 
purpose. 


Methodology 

At JSC, the Search Usability Scale (SUS) was used to determine the usability of the intranet 
search interface, from the user’s perspective. Users were asked to voluntarily respond to the 
survey during the 3 week period of activity. Once the survey was closed, the responses were 
tabulated and scored based on the SUS methodology. The SUS is an ideal tool for collecting data 
because it does not have the same time, equipment, and personnel restraints as more traditional 
evaluations such click-through or eye tracking studies. By participating in an online survey, users 
were able to quickly release their results while continuing to do their work and without interrupting 
the work of others, who are often required to serve as moderators for traditional studies. 

Additional characteristics of the SUS are its ease of use and administration, as well as its 
reliability and validity, regardless of sample size. According to the inventor of the SUS it was 
originally a way to arrive at “some objective measures that would demonstrate the value of 
particular changes was fundamental to that justification in order to get Ul changes included in a 
product release (Brooke, 2013). It is intended purely as a tool for determining the 
problematic/agreeable system status. Michaels (2012) determined that the SUS works best for 
problem identification; the need to probe can be further assessed. 

The SUS uses 10 subjective questions and, most commonly, a 5 point Likert scale to gather in 
inputs used to arrive at the usability score (Usability.gov, 2013). Sauro’s (2010) validity study of 
the SUS compared 5 and 7 point scales in consideration of response error. It was reported that 
due to the 10 question limit imposed by the SUS, the 7 point scale is beneficial because there 
was no “interpolating between choices”. For example, when survey participants were unable to 
choose between 2 and 3 points on the 5 point Likert, the additional 2 points provided an level of 
granulation to more precisely reflect user sentiment. Sauro’s work points to an additional SUS 7 
point validation that found “Seven point Likert scales appear to be sensitive enough to record a 
more accurate evaluation of the interface while remaining relatively compact” (Finstad, 2010). 
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Data Analysis 

Scoring the SUS survey is quick, yet provides a strong suggestion to the perceived usability of 
the system by the end user. Each item's score contribution ranges from 0 to 6. For the positively 
worded items the score contribution is the scale position minus 1. For the negatively worded 
items, the contribution is 7 minus the scale position. Multiplying the sum of the scores by 1.66 
returns the overall value of SUS, normalized to a 1 00-point scale, which is generally easier for 
users to understand and to visualize the outcome. For additional elucidation the results were 
overlaid with an adjective description scale and grading scale. Both of these scales have been 
researched and found to highly correlate with the SUS scale. 
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Figure 1 SUS Scales 


The histograms below represent the grouping of the users calculated SUS scores. Each 
histogram shows the same values with a different method of description, summary statistics, 
adjective rating, and grade scale. 


Histogram of JSC User’s SUS Scores 
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Figure 2 SUS Score Histogram with Statistics 


Min = 31.67 

1 st Quar. = 51.68 

Mean = 59.86 

3 ra Quar. = 67.93 

Max = 83.35 

Std. Deviation = 12.10 

Standard Error = .04 


Table 1 Summary Statistics 
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Histogram of JSC User’s SUS Scores 


<o — | Wc rst Pc or 


13 

= 


20 


n 

40 


Mean 


Gcod Excellent 


60 


I 

80 


SUS Score 

Figure 3 SUS Score Histogram with Adjective Rating 


Best 


lOO 


Histogram of JSC User's SUS Scores 
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Figure 4 SUS Score Histogram with Grade Scale 


Studies indicate the overall mean of SUS surveys have remained constant at about 70. In each 
depiction above a majority of the users give below average ratings towards their satisfaction of 
the search system. In the first histogram the 3 rd quartile was at 67.93. This shows 75% of the 
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users scored the JSC Search system at 67.93 or lower. In the adjective rating, 75%, of the 
responses were below “Good”. On the positive side, 67% of this total fell between “Ok” and 
“Good”, with the reminder, 33%, below “Ok”. The grading scale, however, has the weakest 
representation. In this depiction, 50% of the respondents gave a grade of “F” to the usability of 
the search system at JSC. Additionally, of the remaining 50%, most of them graded JSC Search 
between “C” and “D”, with only a few respondents giving a grade of “B”. 

The data shows most survey participants are not satisfied with the current search environment. 
The next phase in this process is to determine the underlying causes and develop possible 
solutions. 


JSC User Comments 

The first step in identifying root causes is to gather a list of issues or pain points from the users. 
We need to understand the symptoms of the pain in order to prescribe a cure. To that end, one 
question was added to the end of the survey. The respondents were asked, “What would make 
JSC search ideal for you?” The information from this question was to be used as a starting point 
for the next phase of the case study, where we look more closely at the issues affecting search 
and the areas users want improvement. The responses, provide useful information to help 
understand the mindset users are experiencing. Forty-eight of the 71 respondents chose to leave 
a comment. Each of the comments were read and categorized together into common themes. 
High-level categories were created for each group and assigned to the comment. The figure 
below displays a Pareto chart of the categorized comments. Close to 40% of the comments 
pertained to the return inadequate results. The next 30% dealt with unintuitive search interface, 
unfamiliarity with search features, and the delivery of an unexpected information type, i.e. a web 
site instead of a document. In total, 70% of the users issues could be at attributed to these four 
areas, providing a useful starting point as we progress to the next phases. 


Pareto Chart Analysis of Issues from JSC SUS Survey 



Figure 5 Pareto Chart of Issues 
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Next Steps 

Understanding the level of the situation is the first step in defining a long term solution. The 
information collected in this survey provides the basic information to develop a course of action. 

In the next phase of the study, the Knowledge Management Office in collaboration with the 
Information Resources Directorate will research users tendencies and requirements as it pertains 
to search habits. The ultimate goal is to design and implement a search environment based on 
the user’s discovery needs. We need to understand what they are looking for, to deliver the most 
relevant data in a format responsive to their queries. 


Summary 

JSC users were surveyed to determine the System Usability Scale score of the JSC search 
system. Sufficient responses were received to statistically infer findings back to the population. 
The SUS, in use for over 25 years, has been extensively researched and developed to quickly 
and accurately determine the usability of a system. Results of the scores suggest 75%, ±0.04, of 
the population rank the search system below average. In terms of a grading scaled, this equated 
to D or lower. It is obvious JSC users are not satisfied with the current situation, however they are 
eager to provide information and assistance in improving the search system. A majority of the 
respondents provided feedback on the issues most troubling them. This information will be used 
to enrich the next phase, root cause analysis and solution creation. 

For information on this report, contact: 

david.meza-l@nasa.gov 
David Meza, M.S. 

Operational Excellence Program Manager 
Chief Knowledge Office 
NASA - Johnson Space Center 
281.483.6711 

or 

sarah.berndt@nasa.gov 

Sarah Berndt 

JSC Taxonomist 
B1 :878G / Mailcode NA141 
Johnson Space Center, Houston TX 
281-483-3940 Office 
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